Towards Systematic Grammar Pro lingTest Suite Technology Ten Years After Stephan

An experiment with recent test suite and grammar (engineering) resources is outlined: a critical assessment of the EU-funded tsnlp (Test Suites for Natural Language Processing) package as a diagnostic and benchmarking facility for a distributed (multi-site) large-scale hpsg grammar engineering eeort. This paper argues for a generalized , systematic, and fully automated testing and diagnosis facility as an integral part of the linguistic engineering cycle and gives a practical assessment of existing resources; both a exible methodology and tools for competence and performance prooling are presented. By comparison to earlier evaluation work as reeected in the Hewlett-Packard test suite data, released exactly ten years before tsnlp, it is judged where test-suite-based evaluation has improved (and where not) over time. 1 Motivation ...] the study and optimisation of uniication-based parsing must rely on empirical data until complexity theory can more accurately predict the practical behaviour of such parsers. ...] It seems likely that imple-mentational decisions and optimisations based on subtle properties of speciic grammars can ...] be more important than worst-case complexity. Contemporary lexicalized constraint-based grammars (e.g. within the hpsg framework) with wide grammatical and lexical coverage exhibit immense conceptual and computational complexity; as the grammatical framework aims to eliminate redundancy and factor out generalizations, the interaction of lexicon and phrase structure apparatus can be subtle and make it hard to predict how even modest changes to the grammar aaect system behaviour. Additionally, in a distributed grammar engineering setup (i.e. for a project where several Part of the research reported presently was funded by the German National Science Foundation (DFG) within the Special Research Divison 378 (Resource-Adaptive Cognitive Processes) project B4 (perform) and by the German Federal Ministry of Education , Science, Research, and Technology (BMBF) in the framework of the VerbMobil project under grant FKZ:01IV7024. people or even sites contribute to a single grammatical resource) it becomes necessary to assess the impact of individual contributions, regularly evaluate the quality of the overall grammar, and compare it to previous versions. Besides concise coverage (i.e. competence) judgments , in most application scenarios eeciency and resource consumption play an increasingly important role; hence, processing components typically provide a (potentially) large inventory of control parameters and preference settings. When tuning the analysis component to improve system performance , grammar writers often rely on introspec-tion, knowledge of the grammar, and personal experience ; yet, without systematic prooling and performance analysis, processor optimization amounts to guessing …